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20.  personnel.  Data  were  presented  as  a function  of  different 
target  types  and  different  combinations  of  audio  and  video 
displays.  The  third  report  was  concerned  with  the  use  of 
an  auditory  memory  device,  developed  under  the  contract, 
designed  to  aid  the  long-term  memory  of  sonar  operators  for 
recognizing  various  classes  of  targets.  The  fourth  report 
addressed  the  problem  of  perceptual  errors  in  classifying 
sonar  signals  as  a function  of  operator  expectations  about 
the  nature  of  the  target.  The  fifth  technical  report  described 
the  use  of  a small  computer,  in  conjunction  with  an  audio  tape 
library,  to  provide  both  basic  and  advanced  training  in  sonar 
signal  recognition  skills.  The  results  of  two  experimental 
studies  using  this  Small  Computer  Auditory  Training  System 
are  also  described. 
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Introduction 

The  primary  purpose  of  every  indicator  or 
recorder  used  in  sonar  is,  in  some  way  or  other, 
to  make  the  signal  perceptible  to  the  observer... 
the  operation  of  any  machine  is  a joint  enter- 
prise in  which  the  instrument  is  one  partner  and 
the  observer  the  other. 

It  is  essential  that  the  sonar  designer  be 
informed  as  to  the  characteristics  of  observers 
as  well  as  to  the  characteristics  of  the  trans- 
mitting medium.  Sonar,  then,  is  bounded  on  one 
side  by  oceanography , and  on  the  other  by  psycho- 
physiology. ..  . (Horton,  The  Fundamentals  of 
Sonar , 1959.) 

The  purpose  of  the  research  summarized  in  this  final 
report  has  been  to  study  the  performance  of  sonar  operator 
personnel  in  their  most  difficult  of  tasks,  signal  inter- 
pretation or  targot  classification,  as  it  is  affected  by  a 
variety  of  display  variables  and  operational  procedures. 

The  sonar  signals  produced  by  underwater  targets,  whether 
the  result  of  echo  ranging  (active  systems),  or  the  mechan- 
ical activity  of  the  target  itself  (passive  systems),  are 
among  the  most  complex  stimuli  that  man  has  been  asked  to 
detect  ami  interpret.  In  the  early  days  of  sonar  systems, 
the  oar  was  the  primary  analyser  of  these  signals.  With 
the  advent  of  more  sophisticated  sign.l  process. ng  and  dis- 
play techniques,  the  eye  has  also  assumed  a highly  important 
role  in  this  process.  In  fact,  in  some  of  today  s systems, 
visual  pattern  recognition  is  the  sole  means  of  analysis; 
in  others,  aural  and  visual  analyses  play  complementary  roles. 
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.It  was  our  objective  to  study  the  performance  of  sonar 
personnel,  primarily  those  aboard  submarines,  in  reaction  to 
both  audio  and  video  presentations  of  a wide  variety  of  rep- 
resentative target  signals..  In  this  process,  and  in  accor- 
dance with  Horton's  suggestion,  we  tried  to  identify  some  of 
the  variables  associated  with  both  the  man  and  the  machine 
that  affect  the  accuracy  of  sonar  target  classification. 

Clue  Recognition  and  Display  Techniques 

The  first  study  conducted  under  this  contract  was  an 
historical  review  of  research  on  the  perceptibility  of  target 
classification  clues  as  a function  of  a variety  of  display 
techniques  that  had  been  developed  for  active  sonar  systems. 
In  this  review,  we  noted  that  the  task  of  classification  by 
aural  analysis  had  become  increasingly  difficult  and  that 
the  visual  displays  then  in  use  provided  very  limited  inter- 
pretive clues  because  they  had  been  designed  primarily  as 
detection,  rather  than  classification,  displays.  A number 
of  display  techniques  were  described  that  had  been  shown  to 
be  useful  for  classification,  but  had  not  found  their  way 
into  the  design  of  sonar  systoms  that  were  then  operational. 
The  focus  of  dosign  attention  in  active  sonar  systems  had 
been  strongly  on  improved  detection  performance,  but  it 
seemed  to  us  that  the  failure  to  attend  to  the  classifica- 
tion aspect  of  the  problem  might  well  more  than  offset  any 
gain  in  detection  performance  by  creating  a high  false  alarm 
rate.  This  prediction  was  certainly  borne  out  by  the  perfor- 
mance of  several  systems  that  were  then  becoming  operational. 
It  is  heartening  to  note  that  several  sonar  systems  that  are 
now  finding  their  way  into  the  fleet,  seven  years  after  this 
report  was  issued,  reflect  considerably  more  attention  to 
the  display  requirements  for  target  classification  than  was 
true  at  the  time  this  study  began.  More  progress  appears  to 
have  been  made  in  this  respect  with  passive  sonar  systems 
than  with  active  ones,  although  sensitivity  to  the  problem 
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is  reflected  in  the  designs  of  the  most  recent  systems  of 
both  types. 

In  our  first  technical  report,  wc  also  commented  on  the 
evident  interest  in  automating  the  target  classification  pro- 
cess and  reducing  the  operator's  role  to  that  of  a simple 
monitor.  We  pointed  out  that  some  costly  attempts  to  do  this 
in  active  sonar  systems  had  proved  most  disappointing  because 
of  the  very  difficult  problem  of  modeling  the  adaptive  aural 
and  visual  processes  of  man.  This  remains  the  situation  today 
in  active  systems  although  there  has  been  some  success  in 
automating  passive  sonar  classification.  But  there  have  been 
disappointments  with  passive  systems  also,  largely  because  of 
the  temptation  to  replace  man  with  the  computer  rather  than 
use  the  computer  to  augment  his  unique  perceptual  skills. 

In  our  first  report,  we  pointed  out  that  "man's  memory 
and  evaluative  capabilities  for  complex  events  are  admittedly 
fallible;  they  must  be  aided  by  intelligently  designed,  spe- 
cial purpose  devices  and  displays.  But,  somehow,  advantage 
must  be  taken  of  man's  unique  perceptual  flexibility  to  avoid 
undue  reliance  on  the  necessarily  rigid  criteria  of  machines 
for  the  extraction  of  meaningful  signal  characteristics.  The 
consequences  of  failure  to  do  this  will  bo  missed  targets, 
unacceptable  false  alarm  rates,  and  an  increasing  number  of 
'witch-hunts'."  It  is  encouraging  that  a number  of  truly 
interactive  systems  have  recently  been  built  which  reflect  a 
much  more  sophisticated  viewpoint  toward  the  solution  of  this 
problem  than  that  which  prevailed  when  this  project  began. 

This  early  review  of  sonar  displays  and  target  clue  rec- 
ognition was  reported  by  Mackic  and  Parker  in  Technical  Report 
776-1:  "Active  sonar  classification  and  clue  recognition: 

Operator  performance  and  display  enhancement  techniques  (U) 
Human  Factors  Research,  Inc.,  December  1S)C»7  (CONFIDENTIAL) . 


Classification  Performance  of  Submarine  Sonar  Operators 

Our  second  study  represented  the  first  comprehensive 
attempt  to  measure  the  target  classification  performance  of 
submarine  sonar  personnel,  to  measure  their  accuracy  in  rec- 
ognizing clues  presumed  to  be  significant  for  classification, 
and  to  analyze  the  effects  of  audio  and  video  displays  on 
their  performance,  both  singly  and  in  combination. 

Among  the  important  results  of  this  study  was  the  iden- 
tification of  very  extensive  individual  differences  among 
submarine  sonar  personnel  in  ability  to  classify.  These 
differences  were  associated  with  the  amount  of  their  opera- 
tional experience,  but  by  no  means  perfectly,  and  suggested 
that  considerable  improvement  in  the  average' level  of  fleet 
performance  was  possible  through  more  extensive  training  or 
more  frequent  refresher  training. 

Ce-  tain  techniques  for  modifying  the  presentation  of  the 
auditory  stimulus  were  shown  to  enhance  clue  recognition, 
although  greater  accuracy  in  clue  recognition  did  not  always 
translate  into  significantly  higher  classification  performance 
scores.  It  was  shown,  however,  that  performance  was  better 
when  both  audio  and  video  presentations  of  the  target  signals 
wore  used  than  whon  audio  alone  was  used.  The  fact  that  im- 
proved clue  recognition  did  not  always  result  in  more  accurate 
target  classification  suggested  that  there  might  be  deficien- 
cies in  the  process  of  relating  observed  clues  to  a logical 
classification  conclusion.  This  hypothesis  subsequently  was 
tested  and  verified  by  the  Navy  Personnel  Research  and  Develop- 
ment Center  (Abrams  and  Kinchell,  1972)  who  showed  that  the 
use  of  a decision  aid  in  the  form  of  a logic  tree  signifi- 
cantly improved  operator  classification  performance. 

The  video  display  used  at  the  time  of  this  study  had  some 
important  limitations  in  its  presentation  of  target  clues. 

This  was  a systems  problem  attributable,  in  part,  to  certain 
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limitations  in  signal  processing.  In  the  time  since  this 
study  was  reported,  significant  improvements  have  been  made 
in  the  sensor  as  well  as  in  signal  processing  and  display 
techniques.  The  probable  consequence  is  greatly  improved 
clue  recognition  and  classification  performance  although, 
to  date,  we  know  of  no  comprehensive  study  to  determine  ob- 
jectively the  extent  to  which  this  may  have  occurred. 


This  study  also  provided  evidence  concerning  those  target 
classes  that  were  relatively  easy,  or  difficult,  for  fleet 
operators  to  classify.  Considerable  confusion  was  shown  to 
occur  in  the  recognition  of  threat  and  non-threat  classes  of 
•targets.  The  result  of  these  confusions  could  readily  be  a 
high  false  alarm  rate  or,  at  the  least,  time  consuming  dis- 
tractions that  could  interfere  with  mission  success.  (For  an 
excellent  review  of  recent  fleet  experience  reflecting  this 
problem,  see  Becken,  1974.)  To  some  extent,  this  problem 
reflected  deficiencies  in  training.  However,  a more  funda- 
mental problem  was  the  overlapping  signal  space  between  tar- 
gets that  are  threatening  and  those  that  are  not.  The  extent 
of  this  problem  clearly  cannot  be  assessed  except  on  the  basis 
of  a very  comprehensive  target  data  base.  A sufficient  data 
base  did  not  exist  at  the  time  this  study  was  done  and  we 
suspect  it  doos  not  today. 

As  a result  of  this  study,  we  recommended  that  the 
classification  performance  skills  of  all  submarine  sonarmen, 
supervisors,  and  instructors  should  be  determined  by  objec- 
tive performance  tosts  at  loast  once  a year.  We  further 
suggested  that  advancement  in  rate  should  be  made  contingent 
in  part  on  satisfactory  classification  performance  test 
scores.  To  our  knowledge,  neither  of  these  recommendations 
has  been  implemented  although  the  need  seems  greater  than 
over.  Recommendations  that  have  been  implemented  were  that 
classification  training  be  modified  to  emphasise  the  correla- 
tion of  audio  and  video  clues  and  that  target  classification 
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procedure  incorporate  the  use  of  the  video  display  which,  at 
the  time  of  this  study,  was  seldom  used. 

This  research  was  reported  by  Mackie,  Parker,  and  Dods 
in  Technical  Report  776-2:  "Classification  performance  of 

submarine  sonar  operators  (U),"  Human  Factors  Research,  Inc., 
September  1968  (SECRF.T) . 

Memory  Aiding  and  Target  Classification 

The  classification  of  sonar  signals  is  often  dependent 
upon  memory  for  complex  patterns  of  sound  that  the  typical 
operator  may  not  have  experienced  for  months,  or  even  years. 

The  sonar  technician  typically  receives  elementary  training 
in  target  classification  during  basic  operator  training  and 
may  or  may  not  receive  systematic  reinforcement  of  the  re- 
lated perceptual  and  judgmental  skills  after  he  goes  to  sea. 

The  amount  of  reinforcement  differs  widely,  depending  upon 
the  class  of  submarine  to  which  the  sonarman  is  assigned  and 
its  mission.  However,  even  sonar  personnel  who  frequently 
analyze  sonar  contacts  may  suffer  from  lack  of  feedback  con- 
cerning the  true  nature  of  the  target  since  many  targets  are 
never  positively  identified.  This  can  lead,  of  course,  to 
the  reinforcement  of  perceptual  or  judgmental  errors  if  the 
operator  assumes  his  classification  is  correct  when  fact 
it  is  not. 

Because  of  this  lack  of  systematic  reinforcement,  and 
because  the  tasks  of  clue  recognition  and  clue  correlation  are 
so  complex,  it  seemed  to  us  that  a system  which  could  serve  to 
refresh  the  memory  of  sonar  operators  concerning  characteris- 
tics of  signals  typically  produced  by  various  classes  of  tar- 
gets might  be  beneficial  to  performance.  Consequently,  the 
next  study  wo  undertook  was  to  develop  and  test  an  experimental 
auditory  memory  aid  for  passive  sonar  target  classification. 

The  memory  aid  comprised  a 20-channel,  random  access,  magnetic 
tape  playback  deck  loaded  with  recorded  examples  of  signals 
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from  several  classes  of  targets  to  which  an  unknown  contact 
might  belong.  This  provided  a convenient  means  of  comparing 
the  signal  pattern  of  an  unknown  target  with  any  of  the 
selected  examples  contained  in  the  memory  bank. 

To  test  the  effectiveness  of  this  device,  the  classifica- 
tion performance  of  a group  of  experienced  sonar  technicians 
was  tested,  with  and  without  use  of  the  memory  aid.  It  was 
shown  that  the  performance  of  sonar  personnel  with  aided  memory 
was  significantly  better  than  with  unaided  memory;  that  memory 
aiding  was  more  effective  for  some  target  classes  than  others; 
and  that  subjects  who  performed  below  average  in  the  unaided 
condition  benefited  more  from  memory  aiding  than  those  who 
were  above  average.  Since  we  had  developed  only  a relatively 
crude  memory  aiding  device,  with  a very  limiced  store  of  tar- 
get reference  signals,  we  concluded  that  a more  refined  memory 
aiding  system  might  have  even  more  beneficial  effects. 

So  far  as  we  know,  no  steps  have  been  taken  to  incorpo- 
rate a memory  aiding  feature  into  any  sonar  system  presently 
under  development.  This  may  reflect  a de-emphasis  of  the 
audio  display  and  recognition  of  the  inherently  greater  memory 
features  of  video  displays.  We  are  inclined  to  believe, 
however,  that  the  concept  has  general  merit  regardless  of  the 
display  channel;  further,  we  believe  that  there  are  today 
operational  circumstances  under  which  auditory  memory  aiding 
could  significantly  benefit  performance. 

The  research  concerning  memory  aiding  of  sonar  operators 
was  roportcd  by  Dick,  Mecherikoff,  and  Mackie  in  Technical 
Report  776-3:  "Enhancing  passive  sonar  classification  by 

aiding  tho  auditory  memory  of  sonarmen  (U),"  Human  Factors 
Research,  Inc.,  Juno  1970  (CONFIDENTIAL) . 

Stimulus  Error  in  the  Perception  of  Target  Classification  Clues 

Training  sonar  personnel  to  recognize  very  subtle  target 
clues  in  a noise-masked  signal  is  a difficult  task.  liven  the 
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most  proficient  operators  have  difficulty  describing  precisely 
what  characteristics  of  the  signal  they  are  reacting  to  when 
they  "recognize"  a particular  clue.  Certain  classes  of  tar- 
gets are  presumed  to  display  particular  clues  on  the  basis  of 
known  mechanical  features  and  operating  characteristics. 

Whether  or  not  such  clues  are  displayed  is  problematical, 
however,  depending  upon  many  physical  variables  over  which 
the  operator  has  no  control. 

We  were  impressed  with  the  possibility  that  operators 
might  perceive  particular  clues  on  the  basis  of  the  kind  of 
target  they  expected  to  encounter,  since  mission  intelligence 
plays  a significant  role  in  the  total  classification  analysis. 
(For  operational  evidence  of  this  problem,  see  Becken,  or. 
cit.)  We  had  observed  that  target  classification  instructors 
often  appeared  to  perceive  clues  in  recorded  target  signals 
that  were  difficult  for  others  to  discern.  Since  the  instruc- 
tors knew  the  nature  of  the  signal  source,  the  possibility 
existed  that  they  wore  "hearing"  certain  clues  whose  actual 
presence  might  be  doubted.  This  is  a phenomenon  simile*-  to 
"stimulus  error"  that  hcs  long  been  tho  subject  of  study  by 
experimental  psychologists;  it  was  described  as  early  as  1909 
by  Titehoner  (Woodworth,  1938).  Stimulus  error  refers  to  the 
fact  that  an  observer’s  report  of  a perceptual  experience  can 
be  influenced  by  his  knowledge  or  beliefs  about  the  object  he 
is  observing.  His  perceptions  of  an  object  (stimulus)  may  be 
influenced  by  a former  association,  by  the  general  context  in 
which  it  lies,  or  by  unrefleclive  interpretations  (English 
and  English,  1958) . 

Because  of  the  potential  significance  of  such  a phenomenon 
fov  sonar  target  classification,  a study  was  performed  to  test 
whether  the  perception  of  target  clues  (and  therefore  classi- 
fication accuracy)  might  be  influenced  by  knowledge  or  sup- 
positions about  the  nature  of  the  target.  Experienced  sonar 
personnel  listened  to  recorded  signals  fro*  a variety  of  sonar 
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contacts  and  reported  the  clues  they  perceived.  For  half  of 
the  signals,  valid  suggestions  were  made  about  the  nature  of 
the  target;  and  for  the  other  half,  invalid  but  plausible 
suggestions  were  made.  The  operators,  of  course,  had  no 
knowledge  concerning  the  validity  of  these  suggestions. 

The  results  showed  that  the  perceptions  of  target  clues 
by  experienced  operators  were  indeed  influenced  by  the  sug- 
gestions made  about  the  nature  of  the  target;  further,  the 
frequency  with  which  particular  clues  were  reported  was  a 
function  of  certain  stereotypes  associated  with  the  different 
target  classes.  It  was  also  noted  that  judgments  of  whether 
a target  was  a typical  representative  of  a class  seemed  to 
depend  upon  unknown  characteristics  of  the  signal  itself,  and 
to  be  independent  of  whether  or  not  the  target  class  under 
consideration  by  the  operators  was  the  correct  one. 

The  findings  of  this  study  have  clear  implications  for 
instructional  procedures  in  sonar  target  classification  as 
well  as  for  operating  practice  aboard  ship.  Since  the  study 
was  limited  to  aurally  presented  sonar  signals,  it  is  not 
known  whether  visual  presentation  of  the  same  signals  might 
have  reduced  the  stimulus  error.  We  are  inclined  to  think, 
however,  that  this  problem  is  a pervasive  one  in  surveillance 
systems.  Man  has  unique  capabilities  to  perceive  subtle 
characteristics  of  signals  in  noise,  a capability  that  is 
hard  to  emulate  through  computer  software.  At  the  same  time, 
he  is  susceptible  to  perceptual  error  as  a result  of  his  ex- 
pectancies, beliefs,  or  desires  concerning  the  possibility  of 
encountering  a target  of  interest.  Therefore,  we  believe 
that  this  demonstration  of  a fundamental  finding  from  experi- 
mental psychology,  in  the  context  of  the  classification  task, 
has  considerable  operational  significance. 

We  are  net  aware  of  the  extent  to  which  the  results  of 
this  study  may  have  influenced  current  training  procedures. 
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but  are  inclined  to  feel  that  the  impact  has  been  minimal. 
Perhaps  this  again  reflects  the  increased  emphasis  on  video 
displays  in  sonar  target  classification  but,  as  suggested 
previously,  we  expect  that  the  problem  may  well  reside  in 
that  information  channel  as  well. 

The  research  concerning  stimulus  error  was  reported  by 
Dick,  Mecherikoff,  and  Mackie  in  Technical  Report  776-4: 
"Susceptibility  of  experienced  sonarmen  to  stimulus  error  (U)," 
Human  Factors  Research,  Inc.,  August  1970  (CONFIDENTIAL). 

Attempts  To  Define  Perceptual  Dimensions  of  Sonar  Signals 

Sonar  signals  are  commonly  assumed  (by  sonar  personnel) 
to  have  distinctive  sounds  associated  with  the  physical  or 
behavioral  characteristics  of  the  class  of  targets  from  which 
they  come--sounds  which  can  be  reliably  named.  Some  of  the 
terms  describing  the  sounds  relate  directly  to  physical  vari- 
ables, for  example,  tones.  Othors  are  decidedly  subjective 
impressions;  for  instance,  cargo  ship  signals  are  felt  to  be 
more  "laboring,"  less  "galloping,"  lower  in  pitch,  and  loss 
"charging"  than  are  warship  signals.  Descriptive  terms  of 
both  types  play  important  roles  in  both  the  instructional 
process  and  in  operating  doctrine  for  passive  sonar  classifi- 
cation. Such  descriptive  terms  are  often  used  in  training, 
where  the  assumption  is  made  that  the  trainee's  attention  can 
bo  directed  to  certain  qualities  of  the  sound  by  means  of  these 
verbal  pointers.  The  evidence  indicates  that  this  practice 
way  be  deceptive,  sometimes  even  in  the  case  of  terms  describ- 
ing a physically  simple  characteristic  (there  are  adults  who, 
without  specific  training,  do  not  know  how  to  recognize  a 
"high-pitched  tone"  in  a complex  signal).  We  were  interested 
in  how  reliably  these  and  other  descriptive  terms  might  be 
ascribed  to  representative  sonar  signals  and  in  determining 
whether  a move  fundamental  set  of  descriptive  dimensions  for 
the  auditory  experience  might  be  identified.  The  identifica- 
tion of  perceptual  dimensions  potentially  entails  the  reporting 
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of  subjective  experiences  by  the  subjects,  a procedure  usually 
shunned  because  of  its  inherent  unreliability,  or  else  the 
deliberate  variation  of  stimuli  in  simple  ways  along  well- 
defined  physical  dimensions.  The  latter  was  impractical  in 
this  case  because  of  the  complexity  inherent  in  the  sonar 
signal.  To  circumvent  these  problems,  the  method  of  triadic 
comparisons  appeared  on  the  surface  to  be  quite  promising. 

In  this  method,  throe  stimuli  at  a time  are  made  available  to 
the  subject  for  comparison;  the  subject  indicates  which  two 
of  the  three  are  most  similar  and  which  two  are  least  similar. 
From  each  subject's  responses,  a matrix  of  stimulus  similari- 
ties is  constructed  and  any  one  of  a number  of  methods  may  be 
used  to  cluster  the  stimuli  or  factor  the  similarity  matrix, 
v.  f courso,  the  usual  problem  of  the  factor  analytic  methods 
remains:  At  some  point  it  is  necessary  to  reconsider  the 

stimuli  thomselves  and  to  attempt  to  characterize  or  name  the 
dimensions  or  clusters  and  relate  them  to  physical  variables. 

An  additional  problem  with  the  method  of  triadic  com- 
parisons is  that  with  even  a moderate  number  of  stimuli,  the 
number  of  triads  becomes  enormous;  the  demands  on  tho  subject 
become  very  heavy,  and  subject  motivation,  interest,  and 
reliability  eun  suffer.  However,  the  sonar  memory  aiding  de- 
vice, developed  for  earlier  work  under  this  contract,  made  it 
quite  convenient  to  compare  different  signals  so  it  was  decided 
to  proceed  despite  these  recognized  difficulties. 

Four  pilot  studies  were  carried  out  using  the  triadic 
comparison  method  to  test  the  applicability  of  this  method 
for  identifying  perceptual  dimensions  of  passive  sonar  sounds. 
Seven  stimuli  (recordings  of  sonar  contacts)  were  used,  cover- 
ing as  wide  a range  of  target  classes  as  possible.  Naive 
subjects  evaluated  the  35  triads,  and  the  resulting  similarity 
matrices  were  clustered  by  means  of  a hierarchical  clustering 
scheme.  The  resulting  clusters  showed  remarkable  consistency 
across  subjects;  but  in  order  to  check  the  reliability  of  the 
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procedure,  a second  set  of  seven  stimuli  was  selected  and 
administered  to  the  same  subjects.  This  time  there  was  less 
consistency,  the  subjects  complained  that  the  judgments  were 
much  more  difficult,  and  post-experimental  questioning  seemed 
to  indicate  that  part  of  the  lack  of  reliability  was  due  to 
the  fact  that  the  sound  on  a given  tape  loop  was  not  homoge- 
neous throughout  the  entire  loop.  In  an  effort  to  correct 
this  problem,  a 15-second  "consistent-sounding"  segment  was 
taken  from  each  of  the  14  tapes  previously  used,  and  these 
were  administered  in  triads  to  the  subjects.  In  general, 
there  remained  a moderate  amount  of  consistency  across  sub- 
jects in  the  way  the  stimuli  clustered  together,  but  the  reli- 
ability was  not  noticeably  improved  by  using  short  tape  loops. 

The  relationship  of  the  two  or  three  clusters  which 
tended  to  emerge  to  target  classes  or  classification  clues 
was  tenuous.  The  problem,  perhaps  peculiar  to  an  applied 
study  of  this  sort,  stems  from  the  fact  that  the  clusters, 
factors,  or  dimensions  which  may  bo  most  readily  identified 
through  the  analysis  may  not  relate  to  the  categories  of  prac- 
tical interest  (in  this  case,  target  classes);  they  are  factors 
which  arc  recognised  as  irrelevant  by  experienced  operators 
even  without  the  benefit  of  dimensional  analysis.  Some  of  the 
most  outstanding  distinctions  between  passive  sonar  sounds  as 
heard  by  naive  listeners  are  based  on  prepotent  characteristics 
which  happen  to  b*  largely  unrelated  to  target  class,  such  as 
signal-to-noise  ratio,  overall  loudness  of  the  recording,  and 
background  noise  characteristics.  Using  trained  subjects 
(experienced  sonarmen),  however,  introduces  the  genuine  danger 
that  similarity  judgments  will  be  made  on  the  basis  of 
target  cl  an  a , rather  than  r,u  characteristics  of  the  signal. 
Furthermore,  trained  sonarmen,  in  the  formal  i ;.ed  structure  of 
an  experiment  or  test,  will  probably  tend  to  revert  to  the 
"standard"  characteristics  ("naturc-of-sound"  clues)  which 
they  are  supposed  to  use,  even  though  in  practice  their  per- 
ceptual and  decision  processes  may  be  quite  different. 
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Another  fundamental  problem  in  the  use  of  the  triadic 
method  with  very  complex  stimuli  such  as  these  is  that  the 
subject  may  attend  to  different  aspects  of  the  same  stimulus 
at  different  times  and,  of  course,  the  judgments  of  similarity 
and  difference  will  be  seriously  affected.  Subjects  sometimes 
reported  such  an  attention  shift  even  during  their  evaluation 
of  a single  triad.  It  is  also  undoubtedly  true  that  the  char- 
acteristic attended  to  depends  on  the  particular  triad;  for 
example,  "hiss"  might  be  an  outstanding  characteristic  of  one 
of  the  sounds  in  a particular  triad,  but  "hiss"  might  be  com- 
pletely ignored  in  another  triad  if  all  (or  none)  of  the 
sounds  happened  to  have  "hiss."  If  there  are  many  character- 
istics in  each  sound  to  which  the  subject  may  attend,  it  will 
require  a very  large  number  of  stimuli  and  many  replications 
of  the  triads  to  identify  all  the  dimensions,  particularly 
if  the  dimensions  of  practical  importance  are  not  the  most 
obvious  ones.  However,  the  number  of  triads  generated  by 
oven  a rather  small  number  of  stimuli  confronts  the  subjects 
with  an  enormous,  fatiguing  task  and  makes  the  method  pro- 
hibitive. This  is  not  to  imply  that  using  multidimensional 
scaling  or  clustering  techniques  for  the  identification  of 
perceptual  dimensions  in  sonar  signals  is  impossible,  but 
only  that  a successful  effort  to  use  triadic  comparisons  to 
that  end  would  be  massive,  and  would  require  a very  large 
number  of  subjects  each  working  for  many  hours. 

following  this  somewhat  disappointing  experience  with 
triadic  comparisons,  we  decided  to  try  a simpler,  more  direct 
approach.  A list  of  approximately  30  descriptive  words  was 
compiled  by  three  of  tne  research  staff.  Some  of  these  words 
were  taken  directly  from  the  vocabulary  used  in  the  sonar 
aural  analysis  course,  while  others  were  suggested  by  staff 
members  after  listening  to  a great  variety  of  sonar  tape  re- 
cordings. following  each  descriptive  word  in  the  list  was  a 
10-point  scale,  and  the  subjects  were  instructed  to  indicate 
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the  strength  of  that  particular  characteristic  in  the  sound  to 
which  he  was  listening.  The  stimuli  used  were  the  fourteen 
15-second  loops  prepared  previously;  these  were  used  so  that  a 
direct  comparison  could  be  made  with  the  triadic  data.  Again, 
subjects  were  used  who  had  no  sonar  experience;  part  of  the 
objective  was  to  discover  what  kinds  of  terms  naive  subjects 
were  willing  to  use  in  describing  what  they  heard.  More  spe- 
cifically, we  wished  to  find  out  to  what  extent  these  subjects 
would  use  the  vocabulary  regularly  used  in  sonar  training. 

The  results  were  highly  variable  across  the  subjects. 

There  was  little  inclination  among  these  subjects  to  choose 
the  sonar  school  terminology.  Of  course,  it  may  be  argued 
that  one  function  of  the  sonar  training  is  to  attach  the 
proper  verbal  labels  to  the  appropriate  aural  experiences 
and  that  naive  subjects  cannot  be  expected  to  use  the  vocabu- 
lary properly.  Although  this  argument  is  probably  valid,  it 
is  also  true  that  those  terms  are  used  in  training  as  pointers 
to  a particular  characteristic  of  a complex  sound.  The  in- 
structor, having  used  the  terras  for  many  years,  tends  to  over- 
rate the  trainee's  ability  (as  well  as  his  own,  somotimos)  to 
associate  the  terms  with  exactly  the  correct  characteristic. 

There  were  four  terms  in  our  list,  however,  which  seemed 
to  receive  consistently  high  ratings  for  some  stimuli  and  low 
ratings  for  others  (Miss,  High-hitched,  Pulsing,  Squeal),  For 
the  sake  of  a rough  model,  these  four  descriptive  words  were 
treated  as  orthogonal  dimensions,  the  14  stimuli  were  con- 
sidered in  pairs,  and  a hue! id can  distance  was  calculated 
between  the  two  st.'w.li  in  each  pair.  When  the  stimuli  with 
small  distances  t «. > .on  them  were  considered  to  be  clustered 
together,  there  was  good  agreement  between  these  clusters  and 
the  clusters  previously  obtained  through  triadic  comparisons. 
Furthermore,  by  considering  all  14  stimuli  together,  it  was 
possible  to  construct  some  triads  which  had  not  been  previously 
tested,  and  to  predict  on  the  basis  of  the  "distances"  between 
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the  stimuli  how  the  similarity  and  difference  judgments  would 
be  made  within  the  triad.  Although  all  these  data  were  derived 
from  pilot  studies  with  very  few  subjects,  the  consistency  en- 
couraged further  investigation. 

The  above  study  was  then  repeated  using  experienced 
sonarmen,  with  two  changes.  The  stimuli  were  selected  this 
time  in  order  to  represent  each  classification  category  more 
equally,  and  the  response  sheet  was  simplified.  The  results 
essentially  paralleled  those  of  the  first  study:  There  was 

some  distinctiveness  in  the  profile  for  each  target  class 
considered,  but  there  was  too  much  overlap  on  each  "dimension" 
(descriptor)  for  the  technique  in  its  present  state  to  have 
value  as  a classification  aid. 

Thus,,  the  experimenters*  growing  suspicions  that  percep- 
tual dimensions,  in  any  theoretically  or  mathematically  pure 
sense,  would  remain  elusive  wore  strengthened;  while  the 
modest  success  of  the  terminological  study  gave  hope  that 
careful  attention  to  the  processes  of  discrimination,  concept 
acquisition,  and  vorbal  labeling  in  aural  classification 
training  might  yield  a set  of  "dimensions"  which  would  serve 
the  practical  end  of  improved  target  classific  'tion,  and  might 
eventually  bo  related  to  more  fundamental  auditory  processes. 

Because  of  the  exploratory  nature  of  these  studies,  they 
were  not  reported  in  a separate  technical  report.  However, 
they  are  discussed  by  Mcchcrikoff  in  Technical  Report  776-S: 
"Concept  learning  in  the  aural  analysis  of  passive  sonar 
signals,"  Human  Factors  Research,  Inc.,  1974  (CONFllUiXTI AL) . 

tU-c  of  a 1 1 feuy  a lev  in  the  Acquisition  of  Audi  tony  Concepts 
and  the  IHoynooisi  of  Conceptual  Kvrcs'n 

The  failure  for  methodological  reasons  to  discover  the 
perceptual  dimensions  of  sonar  signals  that  relate  to  classi- 
fication performance  led  us  to  try  an  entirely  different 
approach  to  the  problem.  It  is  a reasonable  assumption  that 
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these  dimensions,  or  at  least  their  relevancy,  are  determined 
largely  by  training  and  experience.  Therefore,  another  way  of 
discovering  perceptual  dimensions  might  be  to  manipulate  the 
training  conditions  assumed  to  produce  them.  The  problem  of 
sonar  classification  training  was  viewed  as  one  of  developing 
in  the  trainees  a set.  of  auditory  concepts,  some  related  to 
specific  clues  or  characteristics  of  the  souna,  and  others 
related  to  the  overall  quality  of  the  sounds  of  the  various 
target  classes. 

Very  little  of  the  research  on  concept  acquisition  has 
made  use  of  nonverbal  auditory  stimuli,  and  most  studies  which 
have  used  auditory  stimuli  have  questionable  application  to 
sonar  training  since  they  typically  used  simplistic  stimuli 
in  experimental  sessions  of  very  limited  duration.  We  felt 
that  experimentation  was  needed  which  would  simulate  actual 
training  conditions  (although  in  a highly  controlled  fashion), 
which  would  use  carefully  selected  tape  recordings  of  actual 
sonar  signals,  and  which  would  extend  over  a sufficient  period 
of  time  to  enable  considerable  concept  acquisition  to  take 
place.  To  administer  the  comp 1 ox  experimental  conditions  and 
record  comprehensive  data,  a small  but  sophisticated  computer- 
controlled  auditory  training  system  was  developed  (Small 
Computer  Auditory  Training  Sjstcm). 

Tho  SCATS  consists  of  a small  digitul  computer  (12K  of 
core)  and  magnetic  tape  unit  that  presents  the  stimuli  (sonar 
signals),  administers  the  training  sequences,  and  records  all 
subject  responses  to  the  signals  presented.  Up  to  eight  opera 
tor  terminals  can  be  operated  simultaneously  and  independently 
each  consisting  of  a keyboard,  a limited  visual  display  for 
feedback  and  directions,  and  headphones.  The  taped  audio  sig- 
nals arc  stored  in  endless  loop  cartridges,  and  up  to  S2  high 
fidelity  channels  are  available  instantly  to  any  terminal. 

The  audio  system  has  excellent  fidelity,  the  system  response 
being  flat  from  20-25,000  Ha  1 2 db,  and  the  headphones  from 
20-15,000  lie  * 3 db. 
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Software  development  included  a sophisticated  executive 
program,  similar  in  concept  to  IBM  0/S  360,  and  a flexible 
interpretive  language  designed  for  writing  the  training 
sequences . 

In  addition  to  the  usual  benefits  of  CAI,  the  design  of 
this  system  provided  maximum  efficiency  in  training  audio 
concepts,  provided  an  amount  and  variety  of  carefully  guided 
listening  experiences  many  times  greater  than  that  available 
with  conventional  training  methods,  and  assured  that  the 
trainees'  attention  was  focused  on  the  proper  feature  of  the 
complex  sonar  signal  by  requiring  meaningful  responses  to 
each  stimulus. 

Although  the  system  was  not  designed  as  a training  device 
per  se , it  is  clear  that  it  could  be  used  to  reduce  the  need 
for  instructor  manpower  in  the  laboratory  portions  of  operator 
training.  Its  cost  is  modest  since  it  does  not  require  CRT 
displays,  and  does  not  attempt  verbal  "conversations"  with 
the  trainoes. 

The  basic  functions  of  the  system  arc  as  follows: 

1.  To  prosent  auditory  signals  to  up  to  eight 
operators  working  simultaneously  but  inde- 
pendently; 

2.  To  accept  responses  from  the  operators  via 
a keyboard  terminal  and  evaluate  and  act 
upon  these  responses; 

3.  To  provide  feedback  and  direction  to  the 
operators  through  a dual  display  consisting 
of  a three-digit  numeric  readout  and  a set 
of  If*  lights  indicating  relevant  messages 
printed  on  replaceable  masks; 

4.  To  keep  a comprehensive  log  of  all  stimulus 
conditions  presented  to  the  trainees,  and 
all  trainee  (and  instructor)  actions,  for 
future  detailed  evaluation  of  trainee  per- 
formance and  of  training  effectiveness. 
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Due  to  the  time  required  for  getting  this  small  but 
sophisticated  system  to  function  properly,  and  due  to  an 
unexpected  decrease  in  subject  availability  at  the  Fleet 
ASW  School,  only  two  studies  of  concept  acquisition  were 
eventually  completed  using  this  system.  The  first  study 
was  directed  at  Navy  personnel  who  did  not  have  prior  tar- 
get classification  experience  or  training;  the  second  was 
directed  at  senior  Petty  Officers  at  the  Fleet.  ASM  School 
who  were  experienced,  but  in  general  did  not  consider  them- 
selves experts  in  classification.  In  the  first  instance, 
we  were  interested  in  initial  concept  acquisition;  in  the 
latter,  we  wished  to  identify  the  existence  of  possible 
erroneous  concepts  and  to  prescr?u„,  and  give,  corrective 
training  based  on  individually  diagnosed  deficiencies  in 
performance  (concept  modification). 

In  the  experiment  on  initial  concopt  acquisition,  three 
different  procedures  were  employed: 

1,  Target  Class  Recognition  Training,  In  this 
condition,  examples  of  six  target  classes 
were  presented  as  the  concepts  to  be  learned. 

This  condition  afforded  tho  greatest  opportu- 
nity for  tho  subject  to  respond  to  tho  over- 
all quality  of  the*  sound  and  to  react  to  any 
subtle  patterns  that  might  have  been  common 
to  many  members  of  tho  class  but  which  may  bo 
difficult  or  impossible  to  verbalise.  Of 
the  three  approaches  used,  it  was  the  most 
Gestalt- like. 

2.  Clue  Training  and  Class  Recognition  Training. 

This  condition  most  closely  approximated  the 
kind  of  conceptual  training  currently  given 
in  the  ASK  School.  Attention  was  focused  on 
the  recognition  of  specific  target  clues, 
and  several  clues  were  presented  as  concepts 
to  be  learned.  The  clues  were  related  in  a 
general  way  to  associated  target  classes, 
although  no  specific  logical  procedure  for 
combining  them  into  a specific  classification 
conclusion  was  provided. 
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3.  Clue  Recognition  Training  with  Classification 
* Logic.  In  addition  to  training  on  the  recog- 
nition of  specific  clues,  this  condition 
included  instruction  on  the  use  of  a formal 
decision  logic  which  systematically  took  the 
operator  to  a particular  classification 
conclusion  based  on  his  perception  of  the 
clues.  This  condition  was  the  most  analytic 
and  mechanical  of  the  three  approaches  to 
classification. 

Each  of  the  three  experimental  groups  experienced  a total 
of  15  hours  of  training  and  testing  using  the  SCATS.  Although 
only  three  hours  of  the  training  were  devoted  to  clue  recog- 
nition, the  trainees  made  an  average  score  of  76%  correct  in 
determining  the  presence  or  absence  of  eight  different  clues 
in  representative  sorar  signals.  Some  clues  were  easier  to 
learn  than  others,  the  number  of  correct  judgments  ranging 
from  62%  to  97%.  In  general,  these  percentages  were  about 
the  same  whether  the  clue  was  present  or  absent.  It  is  in- 
teresting to  note  that  the  clue  recognition  scores  of  these 
inexperienced  subjects  were  as  high  after  this  short  period 
of  training  as  those  of  tho  experienced  sonar  Petty  Officers 
tested  in  the  second  study  (68%  to  92%).  This  evidently 
testifies  to  tho  highly  efficient  training  that  SCATS  makes 
possible.  About  75%  of  all  the  training  time  was  spent 
actually  listening  to  sonar  contacts  and  making  perceptual 
judgments. 

The  data  concerning  performance  in  actual  target  classi- 
fication were  less  impressive.  All  three  groups  made  some 
progress  in  learning  to  classify;  but  tho  final  level  of 
achievement  was  only  about  10%  above  chance,  except  for 
Group  3,  which  performed  19%  above  chance.  Since  Group  3 
was  the  only  one  directly  led  through  a structured  logic 
process  for  relating  target  clues  to  a decision  outcome,  this 
result  suggests  that  the  use  of  logic  trees  may  he  one  means 
of  improving  classification  performance.  It  also  confirms 
the  earlier  findings  of  Abrams  and  Winchell  (op,  c 'it.) , who 
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originally  developed  the  logic  tree  used  in  this  study,  in 
their  tests  of  experienced  sonar  operators. 

In  the  second  experiment,  rperienced  submarine  sonar 
personnel  were  first  given  a diagnostic  pretest  by  SCATS,  the 
results  of  which  were  immediately  -"ocessed  to  identify  the 
kind  of  classification  errors  each  operator  had  made.  A pro- 
gram of  remedial  instruction  was  then  embarked  upon,  also 
administered  by  SCATS,  which  was  individually  tailored  to  the 
performance  weaknesses  of  each  sonarman.  Pretest-posttest 
comparisons  showed  that  even  a very  short  (1-day)  refresher 
training  course  had  an  overall  beneficial  effect  upon  target 
discrimination  capability.  The  data  revealed  substantial  in- 
dividual differences  in  the  amount  of  time  operators  used  in 
classifying  the  training  items  and  showed,  in  general,  that 
the  more  items  they  processed  during  refresher  training,  the 
higher  were  their  posttest  scores.  This  fact  alone  argues 
♦for  the  use  of  a system  like  SCATS  to  replace  rather  cumber- 
some procedures  presently  in  use. 

During  remedial  training,  the  computer  routine  required 
the  operators  to  classify  each  sonar  contact  on  four  separate 
occasions.  Each  operator  first  made  a classification  based  on 
his  immediate  impression  after  a 10-second  presentation  of  the 
signal.  Following  this,  the  same  signal  was  turned  on  for  up 
to  5 minutes  for  subjective  analysis  and  reclassification. 
Generally  speaking,  these  responses  were  also  made  quickly, 
usually  in  less  than  one-half  minute.  Following  this,  the 
operator  was  asked  to  judge  the  presence' or  absence  of  eight 
standard  clues,  one  by  one,  and  to  get  a boat  count.  With  the 
results  of  those  analyses  displayed,  he  classified  the  target 
again,  whereupon  tho  correct  beat  count  and  target  clues  were 
displayed.  He  then  classified  the  target  a fourth  time,  taking 
the  corroct  evidence  into  account  if  it  differed  from  his  own. 

It  is  notable  that  classification  performance  on  tho 
training  items  increased  vory  significantly  as  a function  of 
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these  successive  steps,  each  of  which  provided  the  operator 
with  more  information  than  he  had  for  the  preceding  steps. 

The  total  percentages  correct  for  t-.e  four  successive  classi- 
fications were  39%,  44%,  49%,  and  57%  across  all  stimuli  and 
all  subjects.  This.,  showed  that,  on  the  average,  performance 
was  poorest  with  immediate  recognition,  better  with  an  un- 
guided analysis  of  the  target  signal,  better  still  when  the 
operator  was  forced  to  go  through  a systematic  clue  analysis, 
and  best  when  the  correct  clues  were  actually  presented.  This 
last  result  indicates  that  failures  in  clue  recognition,  in 
addition  to  logical  errors,  are  responsible  for  errors  in  tar- 
get classification.; 

There  were  also  some  noticeable  shifts  in  the  recognition 
of  certain  classes  of  targets,  particularly  high  threat  targets, 
with  which  these  personnel  had  had  little  prior  experience. 

For  example,  in  the  case  of  one  important  target  category, 
classification  accuracy  improved  from  18%  to  48%.  This  clearly 
showed  that  the  emphasis  on  this  class  of  target  during  reme- 
dial training  had  a facilitating  effect  on  concept  acquisition 
of  that  particular  class.  Unfortunately,  it  came  at  the  ex- 
pense of  some  loss  in  accuracy  in  classifying  other  types  of 
targets.  (Analysis  indicated,  however,  that  it  was  not  the 
simple  result  of  developing  a response  bias.)  Further  work  is 
necessary  to  determine  what  program  changes  can  be  introduced 
to  achieve  significant  improvement  for  all  target  classes. 
Perhaps  it  is  just  a matter  of  increased  time,  or  better  dis- 
tribution of  training  time,  with  tho  SCATS. 

It  has  already  been  pointed  out  that  a brief  period  of 
clue  recognition  raining  produces  clue  recognition  perfor- 
mance in  novicos  comparable  to  that  of  experienced  sonarmen. 

In  addition,  an  analysis  of  target  class  confusions  for  the 
two  groups  indicated  a substantial  similarity  botweon  novices 
and  experienced  sonarmen  in  the  typos  of  classification  errors 
mado.  In  short,  after  rather  brief  training,  novices  wore 
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beginning  to  behave  in  significant  ways  like  experienced 

* 

operators.  Results  of  this  kind  should  whet  the  appetite 
for  more  extensive  studies  of  the  very  promising  analytic 
and  training  capability  that  computer-based  systems  like 
SCATS  represent.  The  system,  and  detailed  results  of  the 
two  concept  formation  studies,  are  presented  by  Mecherikoff 
in  Technical  Report  776-5  : ’’Concept  learning  in  the  aural 

analysis  of  passive  sonar  signals,"  Human  Factors  Research, 
Inc.,  1974  (CONFIDENTIAL). 

New  sonar  systems  presently  under  development  will  demand 
increased  sophistication,  rather  than  less,  in  the  interpreta- 
tion of  underwater  sound  signals.  The  fundamental  issues 
addressed  in  this  study  will  continue  to  be  important  to  suc- 
cessful operation  of  the  Navy’s  sonar  systems,  though  the 
specific  problems  may  change  as  a function  of  new  display  and 
signal  processing  techniques.  Happily,  there  appears  to  be 
increasing  awa'.ness  throughout  the  Navy  of  the  very  complex 
tasks  faced  by  the  operators  in  the  target  classification 
process.  But  the  need  i~r  continuing  research  and  evaluation 
in  this  problem  area  was  never  more  urgent. 
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